The traditional machine learning has the disadvantage of requiring a large amount of manual annotation to train model and most current transfer learning methods are only applicable to isomorphic space. In order to solve the problems, a new Heterogeneous Compound Transfer Learning (HCTL) method for video content annotation was proposed. Firstly, based on the correspondence between video and image, Canonical Correlation Analysis (CCA) was applied to realize isomorphism of feature space between image domain (source domain) and video domain (target domain). Then, based on the idea of minimizing the cost of projection from this two feature spaces to a common space, a transformation matrix for aligning the feature space of source domain to the feature space of target domain was found. Finally, the features of source domain were translated into the feature space of target target domain by the alignment matrix, which realized the knowledge transfer and completed the video content annotation task. The mean annotation precision of the proposed HCTL on the Kodak database reaches 35.81%, which is 58.03%,23.06%, 45.04%, 6.70%, 15.52%, 13.07% and 6.74% higher than that of Standard Support Vector Machine (S_SVM), Domain Adaptation Support Vector Machine (DASVM), Heterogeneous Transductive Transfer Learning (HTTL), Cross Domain Structural Model (CDSM), Domain Selection Machine (DSM), Multi-domain Adaptation with Heterogeneous Sources (MDA-HS) and Discriminative Correlation Analysis (DCA) methods; while on the Columbia Consumer Video (CCV) database, it reaches 20.73% with the relative increase of 133.71%, 37.28%, 14.34%, 24.88%, 16.40%, 20.73% and 12.48% respectively. The experimental results show that the pre-homogeneous re-aligned compound transfer idea can effectively improve the recognition accuracy in the heterogeneous domain adaptation problems.